Fundus Image Capture System

ABSTRACT

A fundus image capture system operates to determine if the system has captured an image that identifies the fundus, a region of interest in the image, or the current field of view. According to the determination, the system can guide the user to operate the system to capture a desired fundus image.

BACKGROUND

An ophthalmoscope is device used to image the fundus of an eye (or eyefundus) and other structures of the eye. The eye fundus image shows theinterior surface of the eye opposite the lens, including the retina,optic disc, macula and fovea, and posterior pole. This imaging is usedto determine the health of the retina and vitreous humor and to evaluateconditions such as hypertension, diabetic retinopathy, and papilledema.The ophthalmoscope device may include a camera that is used to capturethe images and a display used to display the images obtained by theophthalmoscope device. Medical professionals use the images to diagnoseand treat various diseases.

Typically, the ophthalmoscope is a hand-held device or aheadband-mounted device that is worn by a user, such as a caregiver ormedical professional. To properly capture fundus images, the eye must befirst aligned properly with the device. The user then interacts with thedevice to trigger a capture of an image. For example, the user needs topress a button or select a control on a graphic user interface (GUI) onthe device to capture the image. Misalignment between the device and theeye, or unwanted movement of the device against the eye, can causemislocation of the eye fundus and produce a fundus image of inferiorquality.

SUMMARY

In general terms, this disclosure is directed to a fundus image capturesystem. In one possible configuration and by non-limiting example, thesystem operates to identify a region of interest using a predeterminedimage mask. Various aspects are described in this disclosure, whichinclude, but are not limited to, the following aspects.

One aspect is an apparatus for producing a fundus image. The apparatusincludes a processor, an image capture device for capturing images, anda computer readable storage device storing software instructions that,when executed by the processor, cause the apparatus to: obtain an imagefrom the image capture device; apply a mask to the image to generate amasked image; identify a predetermined contour in the masked image;calculate a centroid of the predetermined contour; and generate guideinformation usable to guide a user to operate the apparatus to capturean image showing a region of interest, the guide information generatedbased on the centroid of the predetermined contour.

Another aspect is a method of capturing an eye fundus image. The methodincludes obtaining an image from an image capture device; applying amask to the image to generate a masked image; identifying a largestcontour in the masked image; calculating a centroid of the largestcontour; displaying the image on a display device; and displaying atarget point on the display device, the target point being arranged at alocation in the image corresponding to the centroid of the largestcontour in the masked image.

Yet another aspect is a computer-readable storage medium comprisingsoftware instructions that, when executed by a processing device, causethe processing device to generate an image mask for identifying a regionof interest in a digital image, wherein the image mask defines a centerregion and a plurality of longitudinal regions, the center region andthe longitudinal regions configured to selectively filter correspondingregions of the digital image, the longitudinal regions extending fromthe center region and being spaced apart from each other withpredetermined gaps.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an example system for producing an eyefundus image.

FIG. 2 illustrates an example fundus image capture system in use on thesubject.

FIG. 3 illustrates an example of the fundus image capture system of FIG.2.

FIG. 4 illustrates an example image capture device.

FIG. 5 is a flowchart of an example method for capturing a desiredfundus image.

FIG. 6 is a flowchart of an example method for determining whether thecaptured image includes a fundus.

FIG. 7 schematically illustrates different images with different averagecolor values and edge characteristics.

FIG. 8 is a flowchart of an example method for identifying a region ofinterest in an image.

FIG. 9 is a flowchart of an example method for guiding a user to track aregion of interest in the image as the user operates the fundus imagecapture apparatus.

FIG. 10 illustrates an example ROI identification mask.

FIG. 11 illustrates an example image and a corresponding masked image.

FIG. 12 illustrates another example image and a corresponding maskedimage.

FIG. 13 illustrates yet another example image and a corresponding maskedimage.

FIG. 14 illustrates yet another example image and a corresponding maskedimage.

FIG. 15 illustrates yet another example image and a corresponding maskedimage.

FIG. 16 is a flowchart of an example method for determining a field ofview of the image.

FIG. 17 shows example images with different fields of view.

FIG. 18 illustrates an example image displayed with a field of view.

FIG. 19 illustrates another example image displayed with another fieldof view.

FIG. 20 illustrates yet another example image displayed with yet anotherfield of view.

FIG. 21 illustrates an exemplary architecture of a computing devicewhich can be used to implement aspects of the present disclosure.

DETAILED DESCRIPTION

Various embodiments will be described in detail with reference to thedrawings, wherein like reference numerals represent like parts andassemblies throughout the several views.

FIG. 1 is a schematic diagram of an example system 100 for capturing afundus image of an eye E. In this example, the system 100 includes afundus image capture system 102, a server computing device 104, and anetwork 106.

The fundus image capture system 102 is used to assist the user U inscreening for, monitoring, and/or diagnosing various eye diseases orother diseases. As described herein, the fundus image capture system 102is configured to capture one or more fundus images of the eye E of asubject, such as a patient. As used herein, “fundus” refers to the eyefundus including the retina, optic disc, macula and fovea, posteriorpole, and other elements. The fundus image is reviewed by a user U forvarious purposes. For example, the fundus image can be used to screenthe subject for eye diseases, or to diagnose or monitor the progressionof various diseases. It will be appreciated that the user that operatesthe fundus image capture system 102 can be different from the user Uevaluating the resulting fundus image.

The user can be a caregiver and medical professional who are experiencedin operating the fundus image capture system. The user can also be anyperson (including the subject him- or herself) who are not familiar withthe fundus image capture system. Capturing a fundus image is not easyfor such users without enough experience of using the fundus imagecapture system, because it is difficult to handle the system to arrangeit against the pupil in the accurate position and orientation. Forexample, when a user uses a handheld fundus image capture device (e.g.,ophthalmoscope), it is difficult to keep track of the center of pupil.In particular, when entering the fundus, the user may get lost due to aslight deviation (such as a few millimeters) off from the pupil center.This situation becomes worse when the subject has a small pupil size. Inaddition, the user may also unintentionally tile the device, resultingin the optical center misaligned with the pupil optical center. Thiscauses losing the field of view of fundus. As described herein, thefundus image capture system 102 is configured to allow unskilled usersto easily track a region of interest in the eye and capture a fundusimage.

In some examples, at least a portion of the fundus image capture system102 is configured to be held by the hand of the user U. In otherexamples, the fundus image capture system 102 is configured to be wornby the subject to capture the fundus image of the subject. An exampleconfiguration of the fundus image capture system 102 is described inmore detail with reference to FIG. 2.

In some embodiments, the system 100 can include the server computingdevice 104 that is in data communication with the fundus image capturesystem 102 via the network 106. The server computing device 104 isconfigured to receive data from the fundus image capture system 102 andperform additional processing and/or storage in an electronic medicalrecord (EMR). In some examples, the fundus image capture system 102 isconfigured to send data associated with the images captured and/orprocessed therein to the server computing device 104, which thenprocesses and store the data for various purposes.

The fundus image capture system 102 and the server computing device 104communicate through the network 106. In one example, the fundus imagecapture system 102 and network 106 are part of a CONNEX™ system fromWelch Allyn of Skaneateles Falls, N.Y., although other systems can beused.

In another example, the server computing device 104 can be a distributednetwork, commonly referred to as a “cloud” server. The fundus imagecapture system 102 communicates with the cloud server throughnon-proprietary, industry standard messaging. Data encryption is alsobased on industry standards.

The network 106 is an electronic communication network that facilitatescommunication between the fundus image capture system 102 and the servercomputing device 104. An electronic communication network is a set ofcomputing devices and links between the computing devices. The computingdevices in the network use the links to enable communication among thecomputing devices in the network. The network 106 can include routers,switches, mobile access points, bridges, hubs, intrusion detectiondevices, storage devices, standalone server devices, blade serverdevices, sensors, desktop computers, firewall devices, laptop computers,handheld computers, mobile telephones, and other types of computingdevices.

In various embodiments, the network 106 includes various types of links.For example, the network 106 can include wired and/or wireless links,including Bluetooth, ultra-wideband (UWB), 802.11, ZigBee, and othertypes of wireless links. Furthermore, in various embodiments, thenetwork 106 is implemented at various scales. For example, the network106 can be implemented as one or more local area networks (LANs),metropolitan area networks, subnets, wide area networks (such as theInternet), or can be implemented at another scale. Additionally, thenetwork 106 includes networks formed between the fundus image capturesystem 102 and a peripheral device using a wireless network protocol(e.g., a mouse connected using Bluetooth, etc.).

FIG. 2 illustrates an example of the fundus image capture system 102 inuse on the subject. In the illustrated example, the fundus image capturesystem 102 is an apparatus configured as an independent handheld devicesuch that the user places the fundus image capture system 102 againstthe subject's face with an end adjacent to or touching the subject'sface surrounding the desired eye socket. In this document, therefore,the fundus image capture system 102 is also referred to as the fundusimage capture apparatus or device 102.

The fundus image capture system 102 includes a physical structureconfigured to house and hold various components of the fundus imagecapture system 102. Such a physical structure can incorporate at leastone of an image capture device, an image process device, and a displaydevice, which are further described in FIG. 3.

In some embodiments, the physical structure of the fundus image capturesystem 102 is configured to be held by the user U for viewing andcapturing images of the fundus. For example, the fundus image capturesystem 102 can be used with PanOptic™ Ophthalmoscope or iExaminer™ fromWelch Allyn of Skaneateles Falls, N.Y.

In other examples, the fundus image capture system 102 includes a mobilecomputing device and an attachment device configured to mount the mobilecomputing device. For example, a user can attach the attachment deviceto an existing mobile computing device, and then take an image of thepatient while monitoring a screen of the mobile computing device. Theattachment device can be configured to be conveniently held by a user sothat the user support the attachment device at a predetermined positionclose to the subject and capture an image as desired. For example, thesystem 102 can be used with PanOptic™ Ophthalmoscope or iExaminer™ fromWelch Allyn of Skaneateles Falls, N.Y. Other examples of the system 102can employ at least part of the disclosure in U.S. patent applicationSer. No. 14/633,601, titled THROUGH FOCUS RETINAL IMAGE CAPTURING, filedFeb. 27, 2015, the disclosure of which is incorporated herein byreference in its entirety. Another example fundus image capture system102 is configured similarly to a device disclosed in U.S. patentapplication Ser. No. 14/557,146, titled DIGITAL COLPOSCOPE SYSTEM, filedDec. 1, 2014, the entirety of which is incorporated herein by reference.

In yet other examples, the physical structure of the system 102 includesa support structure configured to couple to a subject. For example, thesupport structure is an eye glasses frame or a headband. An example ofthe physical structure of this type is disclosed in U.S. patentapplication Ser. No. 14/177,594, titled OPHTHALMOSCOPE DEVICE, filedFeb. 11, 2014, the disclosure of which is incorporated herein byreference in its entirety.

Yet other embodiments of the physical structure are also possible. Insome examples, the physical structure of the fundus image capture system102 includes all mechanical and electrical structures, and hardware andsoftware as a single unit. In other examples, the fundus image capturesystem 102 is a digital ophthalmoscope that wirelessly transfers imagesto a computing device (e.g., mobile computing device or server) forprocessing. Other various examples are also possible.

FIG. 3 illustrates an example of the fundus image capture system 102. Inthis example, the fundus image capture system 102 includes an imagecapture device 110, an image process device 112, a display device 114, aposition measure device 116, and a target guide device 118.

The image capture device 110 is a device configured to capture images ofthe subject, such as the eye E of the subject. In some examples, theimage capture device 110 capture images continuously (e.g., videorecording or burst shots) as the system 102 moves, and some or all ofthe captured images (also referred to herein as image frames) areprocessed and evaluated as described herein. In some examples, themethods described herein are applied to the captured images one-by-oneuntil a desired fundus image is determined to have been obtained. Sincethe methods are performed for each image in a very short period of time(e.g., 30 milliseconds), a plurality of captured images appears to beprocessed in real-time.

In some examples, the image capture device 110 includes a cameraconfigured to capture reflection (e.g., infrared reflection) from anobject to be imaged. An example of the image capture device 110 isillustrated and described in more detail with reference to FIG. 4.

The image process device 112 is a device configured to retrieve one ormore images (also referred to herein as image frames) from the imagecapture device 110 and process the images to obtain a fundus image asdesired. As described herein, the image process device 112 operates toperform image processing to determine that a particular image shows afundus image of the subject as desired. An example of the image processdevice 112 is illustrated and described in more detail with reference toFIG. 5.

The display device 114 is a device configured to display one or moreimages (e.g., one or more fundus images) in original and/or processedformats so that the user reviews the images for diagnosis or monitoring.In some examples, the display device 114 is configured as a touchsensitive display screen and can further present a user interface forenabling the user to interact with the system 102. For example, the userinterface includes graphical control widgets displayed on the displaydevice 114. As described herein, the user interface can include one ormore target guide marks (such as a target point and a track pointdescribed herein) that represent the arrangement (such as positionand/or orientation) of the apparatus 102 relative to a region ofinterest in the subject's eye.

In some examples, the display device 114 is a display of a computingdevice, such as the screen of a mobile computing device (e.g.,smartphones, tablets computers, and other mobile devices).

The position measure device 116 operates to detect the position andorientation of the fundus image capture system 102. In some examples,the position measure device 116 is configured to track the position ofthe image capture device 110 (e.g., a camera thereof) and the opticalaxis orientation. With the fundus tracking information generated asdescribed herein, the position measure device 116 can provide smoothuser feedback to guide a user to move the fundus image capture system102 such that the camera optical axis points to the subject's eye (orfundus).

In some examples, the position measure device 116 includes one or moresensors for determining position and orientation. Examples of suchsensors include a gyroscope, which measures changes in orientationand/or changes in rotational velocity, an accelerometer, which measuresaccelerations to determine changes in velocity and/or changes inposition, and a magnetometer, which measures magnetic fields todetermine absolute orientation. Other types of sensors can also be usedin other examples.

The target guide device 118 operates to provide information for guidingthe user to operate the apparatus 102 to capture a desired image of thesubject's eye. In some examples, the target guide device 118 generatestarget guide marks or icons, and other information of various types(such as visual, audible, and tactile).

In some examples, the target guide device 118 displays visual marks,such as the target point and the track point (as described below) on thedisplay device 114 as the fundus image capture system 102 moves. Suchvisual marks indicate whether the fundus image capture system 102 isarranged to capture a desired fundus image. The target guide device 118is described in more detail with reference to FIGS. 11-15.

In some embodiments, the image capture device 110, the image processdevice 112, the display device 114, and the position measure device 116are implemented in a single computing device. In other embodiments, atleast one of the image capture device 110, the image process device 112,the display device 114, and the position measure device 116 isimplemented in different computing devices.

FIG. 4 illustrates an example of the image capture device 110. In thisexample, the image capture device 110 includes an optical lens element120, an image sensor array 122, an illumination device 124, and acontrol device 126.

The image capture device 110 is configured to capture images of thefundus of the subject. The image capture device 110 is in communicationwith the image process device 112. The images obtained by the imagecapture device 110 can be transmitted to the image process device 112for subsequent processes.

In some embodiments, the images captured by the image capture device 110are digital images. The digital images can be captured in variousformats, such as JPEG, BITMAP, TIFF, etc. In other embodiments, theimages captured by the image capture device 110 are film images. In yetother embodiments, the image capture device 110 includes a digital videocamera. Yet other embodiments of the image capture device 110 arepossible as well.

In some examples, the optical lens element 120 includes a variable focallens, such as a lens moved by a step motor, or a fluid lens (also knownas a liquid lens). Other embodiments of the optical lens element 120 arealso possible.

The image sensor array 122 is a device configured to receive and processlight reflected by the subject's fundus. The image sensor array 122 canbe of various types. For example, the image sensor array 122 is acomplementary metal-oxide semiconductor (CMOS) sensor array (also knownas an active pixel sensor (APS)) or a charge coupled device (CCD)sensor. In some examples, the image sensor array can be configured toreceive and process infrared reflection.

The example image sensor array 122 includes photodiodes that have alight-receiving surface and have substantially uniform length and width.During exposure, the photodiodes convert the incident light to a charge.The image sensor array 122 can be operated in various manners. In someembodiments, the image sensor array 122 is operated as a global reset,that is, substantially all of the photodiodes are exposed simultaneouslyand for substantially identical lengths of time. In other embodiments,the image sensor array 122 is used with a rolling shutter mechanism, inwhich exposures move as a wave from one side of an image to the other.Other mechanisms are also possible to operate the image sensor array 122in yet other embodiments.

The illumination device 124 is a device configured to generate anddirect light towards the eye of the subject when the image capturedevice 110 is in use so that the structures of the eye may be imaged. Insome embodiments, the illumination device 124 includes one or more lightemitting diodes, incandescent bulbs, or fiber optic cables. Theillumination device 124 can also include infrared light sources, such asinfrared LEDs. Yet other embodiments of the illumination device 124 arepossible as well. In some embodiments, the illumination device 124 isconfigured to create multiple illumination conditions, each having adifferent intensity (i.e., brightness) of light. However, someembodiments of the image capture device 110 do not include anillumination device 124.

The control device 126 is a device configured to control the operationof the image capture device 110 to capture images. In some embodiments,the control device 126 is configured to control the optical lens element120, the image sensor array 122, and the illumination device 124. Insome embodiments, the control device 126 is in communication with theimage process device 112.

FIG. 5 is a flowchart of an example method 150 for capturing a desiredfundus image, using the image process device 112. Although the method150 is described to include operations 152, 154, 156, 158, 160, 162,164, and 166, it is also possible in other examples that the method 150includes only some of these operations, and/or additional operationsassociated with the operations described herein.

In one implementation, the steps in the method 150 are implemented as asoftware application running on a computing device, as illustrated inFIG. 21. In some embodiments, the method 150 is developed with OpenCVlibrary. Other libraries can also be used to implement the steps of themethod 150.

The method 150 can begin at operation 152 in which the image processdevice 112 receives an image or image frame from the image capturedevice 110. As described herein, the image capture device 110 cancontinuously capture images while the apparatus 102 is moved by the useragainst the subject's face, eye, or fundus. The image process device 112can receive one or more of the captures images from the image capturedevice 110 in real-time.

At operation 154, the image process device 112 performs a preliminaryprocessing on the image. Various image processing can be performed toenhance fundus tracking.

In some examples, the image can be first converted to a gray scaleimage. A linear filter is then applied to the image using convolution.The linear filter can be configured to remove noise from the image andintensify the contrast of the image. This linear filter operates toimprove fundus tracking by emphasizing edges in the image. For example,when an image is captured in a dark environment, the reflection (e.g.,infrared reflection) is not bright enough to show edges in the imagewith sufficient emphasis to be counted. Further, when the image containstoo much detail, the value of edge detection becomes diluted. Thepreliminary image processing operation improves the functionality offundus tracking.

At operation 156, the image process device 112 determines whether theimage at least partially includes a fundus. As the user moves theapparatus 102 against the subject's face, the image capture device 110of the apparatus 102 can capture anything other than the fundus, such asother portions of the subject's face (e.g., the subject's cheek,forehead, eyebrow, or ear) or any background objects (e.g., desk, table,or wall). At this operation, the image process device 112 determineswhether the captured image actually includes the subject's fundus, andidentify whether the image incorrectly appears to include the subject'sfundus (i.e., a false positive error). An example of the operation 154is described in more detail with reference to FIG. 6.

At operation 158, if it is determined that the image includes a fundus(“YES” at this operation), the method 150 moves on to operation 160.Otherwise (“NO” at this operation), the method 150 returns to theoperation 152 to receive a next image from the image capture device 110and repeat the subsequent operations.

At operation 160, the image process device 112 operates to identify aregion of interest (ROI) in the image. For example, once it isdetermined that the fundus is at least partially identified in thecaptured image (the operation 154), the image process device 112 furtherprocesses the image to locate a ROI in the fundus. An example of theoperation 158 is described in more detail with reference to FIG. 8.

At operation 162, if it is determined that the ROI has been identifiedin the image (“YES” at this operation), the method 150 moves on tooperation 164. Otherwise (“NO” at this operation), the method 150returns to the operation 152 to receive a next image from the imagecapture device 110 and repeat the subsequent operations.

At operation 164, the image process device 112 operates to determine afield of view (FOV) in the captured image. The field of view canrepresent the user's current depth into the fundus. In some examples,the apparatus 102 determines whether the field of view is currentlydesirable and, if not, assists the user to move the apparatus 102 toachieve a desired field of view (e.g., a full field of view). An exampleof the operation 162 is described in more detail with reference to FIG.16.

At operation 166, if it is determined that a desired field of view hasbeen achieved in the image (“YES” at this operation), the method 150continues any subsequent operations. Otherwise (“NO” at this operation),the method 150 returns to the operation 152 to receive a next image fromthe image capture device 110 and repeat the subsequent operations.

FIG. 6 is a flowchart of an example method 170 for determining whetherthe captured image includes a fundus. In some examples, the method 170is performed by the image process device 112 to implement the operation156 in FIG. 5. In the method 170, the image process device 112determines if the fundus has been found as the user manipulates andmoves the apparatus 102 against the subject's face or eye.

In one implementation, the steps in the method 170 are implemented as asoftware application running on a computing device, as illustrated inFIG. 21. In some embodiments, the method 170 is developed with OpenCVlibrary. Other libraries can also be used to implement the steps of themethod 170.

Although the method 170 is described to include the operationsillustrated in FIG. 6, it is also possible in other examples that themethod 170 includes only some of these operations, and/or additionaloperations associated with the operations described herein. For example,some of the illustrated operations are optional depending on theattributes (e.g., size or quality) of the captured image.

The method 170 begins at operation 172 in which the image process device112 receives an image captured by the image capture device 110. Asdescribed herein, in some examples, the captured image can bepre-processed as described in FIG. 6 before the image is processed inthe method 170.

At operation 174, the image process device 112 resizes the image. Insome examples, the image is reduced to a smaller size to decrease thenumber of cycles needed to complete the subsequent processes, such asthe subsequent operations in the method 170. For example, when the fullyscale image has too much detail that are not interested, the fully scaleimage requires more computing resources for image processing than asmaller image. By way of example, the image can be resized to I/O scale.

At operation 176, the image process device 112 blurs the scaled image toremove details that are not interesting. This blurring process is usedto prevent detection of irrelevant edges or minor edges and only tofocus on hard edges, such as the bridge of a nose, the edge of eyelid,the edges of other facial elements, the edges of environmental objectsaround the subject's face, and other hard edges.

At operation 178, the image process device 112 obtains an average colorvalue of the scaled image. The average color value of the scaled imagecan be calculated using various known algorithms.

At operation 180, the image process device 112 performs edge detection.In some examples, the image process device 112 calculates the amount ofedges that are detected in the scaled image. For example, the amount ofedges can be determined as the number of pixels constituting the edges.Other ways to measure the amount of edges are also possible in otherexamples.

At operation 182, the image process device 112 determines whether thescaled image has a predetermined characteristic that is representativeof a fundus in the scaled image. In some examples, the predeterminedcharacteristic is associated with at least one of a color value and anedge characteristic, as illustrated in FIG. 7. For example, the imageprocess device 112 can determine whether the average color value of thescale image, as obtained in the operation 178, meets a color thresholdvalue that is indicative of an image including a fundus. In addition oralternatively, the image process device 112 determines whether theamount of edges (e.g., the number of pixels constituting the edges) inthe scaled image, as obtained in the operation 180, meets an edgethreshold value that is indicative of an image including a fundus.

At operation 184, if it is not determined that the scaled image has thepredetermined characteristic (“NO” at this operation), the method 170moves on to operation 186, in which the image process device 112recognizes that the image does not show a fundus. As such, the image isnot considered to include the fundus to be detected. Then, the imageprocess device 112 can receive another image taken by the image capturedevice 110 (at the operation 152) and perform the subsequent operationsas described herein.

Alternatively, if it is determined that the scaled image has thepredetermined characteristic (“YES” at this operation), the method 170goes to operation 188, in which the image process device 112 identifiesthe image as an image showing a fundus. As such, the image is consideredto include the fundus to be detected, and thus treated as being near orin fundus. Then, the image process device 112 can perform subsequentoperations, such as the operation 160 in FIG. 5.

FIG. 7 schematically illustrates different images with different averagecolor values and edge characteristics. In the illustrated example, afundus 192 is shown in different sizes in different images 190. Thefundus 192 can be identified by an edge or contour 194 and have a color196. By way of example, the size of the fundus 192 is bigger in a secondimage 190B than in a first image 190A. In a third image 190C, theapparatus 102 has moved into the fundus 192. In a fourth image 190D, theapparatus 102 has further moved into the fundus 192. As such, the sizeof the fundus in an image can change the average color value and theedge characteristic of the image. Based on this relationship, theaverage color value and/or the edge characteristic of an image can beused to determine whether the image includes a fundus.

FIG. 8 is a flowchart of an example method 210 for identifying a regionof interest (ROI) in the image. In some examples, the method 210 isperformed by the image process device 112 to implement the operation 158in FIG. 5. In the method 210, the image process device 112 locates a ROIin the image. Once the image has been identified as showing a fundus inthe method 170 (FIG. 6), the image is further processed and evaluated tolocate a ROI in the image. In some examples, the ROI is the center ofthe fundus in the image. In other examples, another location in theimage can be identified as the ROI.

In one implementation, the steps in the method 210 are implemented as asoftware application running on a computing device, as illustrated inFIG. 21. In some embodiments, the method 210 is developed with OpenCVlibrary. Other libraries can also be used to implement the steps of themethod 210.

Although the method 210 is described to include the operationsillustrated in FIG. 8, it is also possible in other examples that themethod 210 includes only some of these operations, and/or additionaloperations associated with the operations described herein. For example,some of the illustrated operations are optional depending on theattributes (e.g., size or quality) of the captured image.

At operation 212, the image process device 112 obtains an image. In someexamples, the image is the originally-sized image that corresponds withthe scaled image processed in the method 170 as described in FIG. 6.

At operation 214, the image process device 112 applies a mask to theimage to remove any non-fundus area. Various known masks can be used forthis purpose.

At operation 216, the image process device 112 erodes the image toremove noises, such as irrelevant or minor features in the image. Atoperation 218, the image process device 112 dilates the image. Atoperation 220, the image process device 112 thresholds the image tofurther segment the image and remove all pixels that are too dark or toobright. The thresholding process is used to separate out regions of theimage corresponding to the fundus to be analyzed. For example, a binarythreshold (e.g., a threshold to zero function in OpenCV) can be used toperform thresholding in this operation. The thresholding process canseparate regions in the image based on the variation of intensitybetween the pixels corresponding to the region of interest and thepixels of background.

At operation 222, the image process device 112 applies an ROIidentification mask 300 (such as in FIG. 10) to the image. The ROIidentification mask is configured to distinguish one or more things thatare not typically regions of interest but end up in certain areasconsistently. Such things include facial portions or elements around oradjacent the eye or fundus, such as eyelid, nose, eyebrow, and forehead.When a glare is found in an image, which resembles the reflection from aregion of interest in the eye, it may be in fact the reflection off ofother regions that are not in interest, such as outside the eye orfundus. As described herein, the ROI identification mask operates toidentify the region of interest, distinguishing it from regions that arenot in interest.

In some examples, the ROI identification mask is configured to break upcontours in regions of the image that may cause false positive errors.As described herein, the ROI identification mask is configured to breakone or more contours in the image into pieces. The image process devicecan then measure the area of each piece, identify the piece with thelargest area as a ROI, and then calculate the centroid of the ROI. Thecentroid of the ROI is mapped to the actual image (or identified in theactual image) and used as a target point, which can be presented in thedisplay device of the apparatus 102. An example of the ROIidentification mask 300 is illustrated in more detail with reference toFIG. 10.

At operation 224, the image process device 112 determines whether theROI is identified in the image. As described above, once the ROIidentification mask is applied, the image process device can measure thearea of each piece, identify the piece with the largest area as a ROI,and then calculate the centroid of the ROI to use it as the targetpoint. In some examples, the ROI is considered to be identified when thetarget point is positioned at a predetermined location in the image oron the display screen of the apparatus 102. An example of this operationis illustrated in more detail with reference to FIG. 9.

At operation 226, the image process device 112 generates and providesguide information to the user of the apparatus 102. In some examples,the guide information is presented via the display device 114 of theapparatus 102. An example of the guide information is illustrated withreference to FIGS. 11-15.

At operation 228, if it is determined that the ROI has not beenidentified (“NO” at this operation), the method 210 moves on tooperation 230, in which the image process device 112 recognizes that theapparatus 102 is not arranged in a desired position to capture a fundusimage. Then, the image process device 112 can receive another imagetaken by the image capture device 110 (at the operation 152) and performthe subsequent operations as described herein.

Alternatively, if it is determined that the ROI has been identified(“YES” at this operation), the method 210 goes to operation 232, inwhich the image process device 112 considers that the apparatus 102 isin a desired position to capture a fundus image. As such, it is regardedthat the image includes the fundus and that the ROI has been identified.Then, the image process device 112 can perform subsequent operations,such as the operation 164 in FIG. 5.

FIG. 9 is a flowchart of an example method 250 for guiding a user totrack a region of interest (ROI) in the image as the user operates thefundus image capture apparatus 102. The method 250 is described withalso reference to FIGS. 10-15. FIG. 10 illustrates an example ROIidentification mask to be applied to track the ROI in the image. FIGS.11-15 illustrate five example images captured and displayed in thedisplay device, and five example diagrams that demonstrate imageprocessing to track the ROI in the image.

As described herein, in some examples, the region of interest is apredetermined location of the fundus, such as the center of the fundusin the image. In some examples, the method 250 is performed by one ormore devices in the apparatus 102 to implement at least one of theoperations 222, 224, 226, 228, 230, and 232 in the method 210, asdescribed in FIG. 8. Although the method 250 is primarily describedbelow to be executed by the image process device 112, it is noted thatother devices of the apparatus 102, such as the display device 114, theposition measure device 116, and the target guide device 118, can alsobe used to perform at least part of the method 250.

In one implementation, the steps in the method 250 are implemented as asoftware application running on a computing device, as illustrated inFIG. 21. In some embodiments, the method 250 is developed with OpenCVlibrary. Other libraries can also be used to implement the steps of themethod 250.

Although the method 250 is described to include the operationsillustrated in FIG. 9, it is also possible in other examples that themethod 250 includes only some of these operations, and/or additionaloperations associated with the operations described herein. For example,some of the illustrated operations are optional depending on theattributes (e.g., size or quality) of the captured image.

The method 250 can begin at operation 252, in which a mask is applied tothe image. In some examples, the mask can be an ROI identification mask300, as illustrated in FIG. 10. The ROI identification mask 300 operatesto distinguish things that are not typically regions of interest, andidentify an ROI.

As illustrated in FIG. 10, the ROI identification mask 300 is configuredto filter the image into one or more segments or pieces. For example,the ROI identification mask 300 breaks up contours in the image intosegments or pieces. By way of example, in FIGS. 11-15, when the ROIidentification mask 300 is applied to the image 330, a diagram 332 thatrepresents data about the masked image demonstrates a plurality ofsegments 350, which are also referred to herein as masked contours 350,or simply as blobs or contours 350. In the masked image or diagram 332,the contours 350 are defined as enclosed, contiguous areas.

In some examples, the ROI identification mask 300 is configured to be inthe same size as images 330 captured and displayed in the apparatus 102.The same ROI identification mask 300 is applied to different images 330captured as the apparatus 102 moves against the subject's face or eye.

In some examples, the ROI identification mask 300 defines a centerfilter region and wedge-shaped filter regions arranged around the centerfilter regions. This configuration ensures that the reflection shown inthe image is from the eye, not from something else.

For example, the ROI identification mask 300 defines a center region 302that selectively filters a corresponding region of the captured image330. In addition, the ROI identification mask 300 includes a pluralityof longitudinal regions 304 that selectively filter correspondingregions of the captured image 330. In some examples, the longitudinalregions 304 are connected to, and extend from, the center region 302. Insome examples, the longitudinal regions 304 extend in parallel and arespaced apart from each other with predetermined gaps. In the illustratedexample, the longitudinal regions 304 extend vertically. In otherexamples, the longitudinal regions 304 can extend in differentdirections.

In addition, the ROI identification mask 300 includes a crossing region306 configured to selectively filter a corresponding region of thecaptured image 330. The crossing region 306 can run across the mask 300.In some examples, the crossing region 306 extends through the centerregion 302. The crossing region 306 is arranged to modify and distributethe masking area as desired. In the illustrated example, the crossingregion 306 runs horizontally. In other examples, the crossing region 306can extend in different directions.

With the crossing region 306, the longitudinal regions 304 are separatedinto two groups, one group of longitudinal regions (e.g., at the upperside of the mask) being spaced apart from the other group oflongitudinal regions (e.g., at the lower side of the mask). With theconfiguration of the center region 302, the longitudinal regions 304 andthe crossing region 306, the target point (or the centroid of thelargest contour) can be moved smoothly. For example, when the user movesthe apparatus 102, the target point 370 appears to move smoothly on thedisplay screen.

The regions of the mask 300, such as the center region 302, thelongitudinal regions 304, and the crossing region 306, are defined by asurrounding portion 308 configured to block corresponding regions of theimage to which the mask is applied.

In some examples, the center region 302, the longitudinal regions 304,and the crossing region 306 are configured to filter the correspondingregions of the image 330 to which the mask 300 is applied, when thepixels of the corresponding regions have predetermined values. Where theimage 330 is a grayscale image, the regions 302, 304, and 306 of themask 300 filter the pixels having values greater than a threshold value.The other portions of the mask 300, such as the surrounding portion 308,block all pixels of corresponding portions of the image.

As exemplified in FIGS. 11-15, the center region 302 of the mask 300 isarranged at a location in which a desired object or element (i.e., theregion of interest) is supposed to be arranged in the image. Incontrast, the longitudinal regions 304 of the mask 300 are arranged at alocation in which objects or elements that are not in interest (i.e.,false positive errors) are typically arranged.

For example, in FIG. 11, the image 330 generally captures regionsoutside the pupil and does not show the region of interest, such as thecenter of fundus. In fact, a contour of reflection 336 at a lower sideof the image 330 shows the eyelid and part of the eyeball. A contour ofreflection 338 at an upper side of the image 330 is actually part of thepupil, and a dark region 340 is the rest of the pupil. As such, thecontours 336, 338, and 340 are generally not regions of interest, andare positioned primarily over the longitudinal regions 304 in the mask300. As shown in the masked image 332, the contours 336, 338, and 340are generally filtered by the longitudinal regions 304 of the mask 300,resulting in multiple small pieces of masked contours 350 in the maskedimage 332.

In contrast, as shown in FIG. 15, the image 330 captures inside of thefundus and shows the region of interest. As shown, the region ofinterest (e.g., the center of fundus) is positioned over the centerregion 302 in the mask 300. Thus, the region of interest is filteredprimarily by the center region 302 of the mask 300, resulting in asingle large contour 350 in the masked image 332.

Referring again to FIG. 9, once the image 330 is masked (operation 252),the image process device 112 obtains contours 350 in the masked image332, at operation 254. As illustrated in FIGS. 11-15, the contours 350are defined as enclosed, contiguous regions in the masked image 332, andare separated from each other. In the illustrated example, the contours350 in the masked image 332 are formed by the center region 302, thelongitudinal regions 304, and/or the crossing region 306 of the mask 300when the mask 300 is applied to the image 330.

At operation 256, the image process device 112 identifies the largestcontour 350. In some examples, the image process device 112 calculatesthe area of each contour 350 in the masked image 332, and comparesbetween the areas of the contours 350 to determine one of the contours350 that has the largest area. In some examples, the area is calculatedby the number of pixels within each contour. In other examples, theareas are determined by relative sizes of the contours. Other methodscan be used to calculate the area of each contour in other examples.

At operation 258, the image process device 112 calculates a centroid 360of the largest contour 350. The centroid 360 indicates a center of massof the largest contour. In some examples, the largest contour is assumedto be of uniform density. Such a centroid can be determined in variousmethods.

By way of example, in FIG. 12, the masked image 332 includes a singlecontour 350 at the upper side and twenty (20) contours 350 at the lowerside of the masked image 332. The image process device 112 woulddetermine the upper-side contour 350 as the largest contour in themasked image and calculate the centroid 360 of the upper-side contour350. The centroid 360 in the other masked images 332 (FIGS. 10 and13-15) is similarly obtained.

At operation 260, the image process device 112 maps the centroid 360 ofthe largest contour 350 to the image 330. The mapping 362 of thecentroid 360 is illustrated in FIGS. 11-15.

At operation 262, the image process device 112 generates and displays atarget point 370 on the display screen of the apparatus 102. Asillustrated, the target point 370 is placed over the image 330 shown inthe display screen. The target point 370 is presented at a location towhich the centroid 360 is mapped. Therefore, the target point 370 canrepresent the centroid of the largest contour, which can correspond tothe center of the region of interest. The target point 370 is used as aguide mark that assists the user to move the apparatus 102 until the ROIis identified (as illustrated in FIG. 15). As illustrated in FIGS.11-15, the target point 370 moves on the display screen as the apparatus102 moves by the user's operation against the subject's face or eye.

At operation 264, the image process device 112 determines whether thetarget point 370 matches a track point 380 on the display screen of theapparatus 102.

Along with the target point 370, the track point 380 is used to guidethe user to move the apparatus 102 to desired position and orientationto capture a desired image. In some examples, the track point 380 isarranged and fixed at a predetermined location of the field of view (andthus the display screen). For example, the track point 380 is arrangedat the center of the field of view (i.e., at the center of the displayscreen). The mask 300 can be applied to each image 330 such that thecenter of the center region 302 in the mask 300 is aligned to correspondto the track point 380 in the image 330.

The track point 380 functions as a reference point to which the targetpoint 370 needs to fit. As the apparatus 102 moves, the target point 370can move because the images captured by the apparatus 102 are differentdepending on the position of the apparatus 102 against the subject.However, the track point 380 is fixed on the display screen of theapparatus 102, and represents a point to which the apparatus 102 shouldbe arranged and oriented. Therefore, the user is required to move theapparatus 102 until the target point 370 matches the target point 380.As such, the target point 370 and the track point 380 are configured toassist the user move, arrange, and orient the apparatus 102 withoutpaying attention to what the user are actually looking at through thedisplay device. The user only needs to keep moving the target point 370into the track point 380.

In the illustrated example, the target point 370 is presented as a solidball or square, and the track point 380 is displayed as a hollow circleor square. Other shapes or characteristics of the target point 370 andthe track point 380 are possible in other examples.

By way of example, in FIGS. 11-14, the target point 370 does not matchthe track point 380, which indicates that the apparatus 102 is notarranged against the subject's eye as desired to identify the ROI, suchas the center of fundus. In contrast, in FIG. 15, the target point 370matches the track point 380, thereby representing that the apparatus 102is appropriately arranged and oriented against the subject's eye, andthat the captured image 330 includes and identifies the ROI as desired.

In some examples, to improve readability, the colors of the target point370 and/or the track point 380 can change depending on the position ofthe target point 370 relative to the track point 380. In one embodiment,the target point 370 and the track point 380 has a first color (e.g.,red) when the target point 370 and the track point 380 are misaligned asin FIGS. 11-14, and change their color to a second color (e.g., green)when the target point 370 is overlapped with the track point 380 asshown in FIG. 15. Other color schemes are also possible. In otherexamples, in addition or alternatively to the color scheme, variousvisual and/or tactile schemes can be used to indicate the matchingbetween the target point 370 and the track point 380. For example, whenthe target point 370 and the track point 380 are aligned, the points 370and 380 can blink, and/or provide tactile feedback (e.g., vibration orhaptic effect) to the user through the apparatus.

Referring still to FIG. 9, at operation 264, if it is determined thatthe target point 370 does not match the track point 380 (“NO” at thisoperation), the method 250 moves on to operation 266, in which the imageprocess device 112 recognizes that the apparatus 102 is not arranged ina desired position to capture a fundus image. Then, the image processdevice 112 can receive another image taken by the image capture device110 (at the operation 152 in FIG. 5) and perform the subsequentoperations as described herein.

Alternatively, if it is determined that the target point 370 matches thetrack point 380 (“YES” at this operation), the method 250 goes tooperation 268, in which the image process device 112 considers that theapparatus 102 is in a desired position to capture a fundus image. Assuch, it is regarded that the image includes the fundus and that the ROIhas been identified. Then, the image process device 112 can performsubsequent operations, such as the operation 164 in FIG. 5.

Referring to FIGS. 11-15, the method 250 is further described, which isto guide the user until the user moves the apparatus 102 to a desiredposition against the subject. In FIGS. 11-15, the images 330 illustrateexample images displayed on the display screen such that the user canmonitor the images being captured by the apparatus 102 while moving theapparatus 102 against the subject. The masked images 332 are exampleresults of applying the ROI identification mask 300 to the correspondingimages 330, respectively.

In FIGS. 11-14, when the image 330 is filtered with the ROIidentification mask 300, one or more contours 350 (e.g., multiplecontours in FIGS. 11 and 12, and one contour in FIGS. 13 and 14) aregenerated as illustrated in the masked image 332. Then, the largestcontour 350 is identified and the centroid 360 is calculated. Thecentroid 360 is then mapped into the image 330, and the mapped locationis represented by the target point 370 in the image 330. As shown, thetarget point 370 is not fit into the track point 380 that is fixed inthe center of the display screen (and thus the center of the image 330).Thus, the user needs to move the apparatus 102 to try to match thetarget point 370 with the track point 380.

As such, the ROI identification mask 300 is used to identify the regionof interest. Without the ROI identification mask 300, for example, theimage 330 in FIG. 12 may be interpreted such that the region of interestis in the lower region of the image when, in fact, the lower region inthe image is the subject's lower eyelid.

In FIG. 15, when the image 330 is filtered with the ROI identificationmask 300, one contour 350 is generated as illustrated in the maskedimage 332. Then, the centroid 360 of the contour 350 is calculated. Thecentroid 360 is then mapped into the image 330, and the mapped locationis represented by the target point 370 in the image 330. As shown, thetarget point 370 matches the track point 380. Thus, the apparatus 102 isappropriately arranged against the subject's fundus, and the image 330contains the fundus and the desired ROI.

FIG. 16 is a flowchart of an example method 270 for determining a fieldof view (FOV) of the image. The method 270 further allows the user tomove the apparatus 102 to achieve a desired field of view. The method270 is described with also reference to FIGS. 17-20. FIG. 17 showsexample images with different fields of view. FIGS. 18-20 illustratethree example images displayed in the display device while the field ofview is adjusted.

In some examples, the method 270 is performed by one or more devices inthe apparatus 102 to implement the operation 164 in FIG. 5. Although themethod 270 is primarily described below to be executed by the imageprocess device 112, it is noted that other devices of the apparatus 102,such as the display device 114, the position measure device 116, and thetarget guide device 118, can also be used to perform at least part ofthe method 270.

In one implementation, the steps in the method 270 are implemented as asoftware application running on a computing device, as illustrated inFIG. 21. In some embodiments, the method 270 is developed with OpenCVlibrary. Other libraries can also be used to implement the steps of themethod 270.

Although the method 270 is described to include the operationsillustrated in FIG. 16, it is also possible in other examples that themethod 270 includes only some of these operations, and/or additionaloperations associated with the operations described herein. For example,some of the illustrated operations are optional depending on theattributes (e.g., size or quality) of the captured image.

Once the apparatus 102 is arranged to view the inside of the fundus, theapparatus 102 can further operate to ensure that the apparatus 102 isarranged appropriately to capture the fundus in a full field of view. Asthe field of view from the apparatus 102 comes into the area of thefundus, the apparatus 102 further operates to assist the user to adjustthe position and/or orientation of the apparatus to make the best fieldof view. The method 270 helps the user achieve the full field of viewwhile the apparatus 102 is not in a desired position or orientation dueto the user's improper operation of the apparatus and/or the subject'ssudden movement.

The method 270 begins at operation 272, in which the image processdevice 112 identifies a largest contour 390 (FIG. 17) in the image 330.As illustrated in FIG. 17, different images 330 (such as 330A-F) havedifferent largest contours 390.

In some examples, the largest contour 390 can be identified byprocessing and evaluating the image 330, independently from the otheroperations described herein. In other examples, the largest contour 390can be identified from one or more of the operations described herein,such as the operations 252, 254, and 256 in the method 250 (FIG. 9). Inyet other examples, the largest contour 390 can be obtained inoperations that are separate from the other operations described hereinbut perform similarly to one or more of the other operations.

At operation 274, the image process device 112 obtains the area of thelargest contour 390. The area of the largest contour 390 can be used torepresent the current field of view of the image, which reveals theuser's current depth into the fundus.

In some examples, the area is calculated by the number of pixels withinthe contour 390. In other examples, the area is determined by a relativesize of the contour 390. Other methods can be used to calculate the areaof the largest contour 390 in other examples.

At operation 276, the image process device 112 compares the area of thelargest contour 390 with a threshold value. The threshold value isrepresentative of a desired field of view in the image. In someexamples, the threshold value is a single predetermined value. In otherexamples, the threshold value is a range of value.

At operation 278, the image process device 112 determines whether thearea of the largest contour 390 meets the threshold value. Where thethreshold value is a range of value, the area can be considered to meetthe threshold value if the area falls within the range of value.

By way of example, if the area (e.g., the number of pixels) of thelargest contour 390 is less than a first value (e.g., 1000 pixels), itis considered that the threshold value is not met, and the image isregarded as not having a desired field of view. For example, the image330F has the largest contour 390 having an area smaller than the firstvalue. This can imply that the image captures completely outside of thefundus.

If the area of the largest contour 390 ranges between the first value(e.g., 1000 pixels) and a second value (e.g., 2500 pixels), it isconsidered that the threshold value is still not met, and the image isregarded as not having a desired field of view. However, it can implythat the field of view from the apparatus is partially moving into thefundus. For example, the images 330D and 330E have the largest contours390 having areas between the first and second values, and thus representthat the field of view has partially entered into the fundus.

If the area of the largest contour 390 ranges between the second value(e.g., 2500 pixels) and a third value (e.g., 6000 pixels), it isconsidered that the threshold value is met, and the image is regarded ashaving a desired field of view. Thus, it implies that the field of viewfrom the apparatus has completely moved into the fundus. For example,the images 330B and 330C have the largest contours 390 having areasbetween the second and third values, and thus represent that the fieldof view has entered into the fundus. In some examples, the apparatus maystill need to slightly adjust its angle to reach the full field of view.

If the area of the largest contour 390 is greater than the third value(e.g., 6000 pixels), it is considered that the threshold is not met, andthe image is regarded as not having a desired field of view. This mayindicate a false positive error, which implies that the apparatus iscompletely outside the fundus. For example, the image 330A has thelargest contour 390 with the area greater than the third value. However,the image 330A is actually an image of the subject's cheek when theapparatus 102 is pressed against it.

Referring still to FIG. 17, if it is determined that the area of thelargest contour 390 does not meet the threshold (“NO” at thisoperation), the method 270 moves on to operation 280, in which the imageprocess device 112 recognizes that the apparatus 102 is not arranged ina desired position to capture a fundus image (e.g., not in a desiredfield of view). Then, at operation 282, the image process device 112(and/or the display device 114 and the target guide device 118) providesguide information to the user to assist the user to move the apparatus102 until the apparatus 102 reaches to its desired position (with thedesired field of view). The image process device 112 can further receiveanother image taken by the image capture device 110 (at the operation152 in FIG. 5) and perform the subsequent operations as describedherein.

Alternatively, if it is determined that the area of the largest contour390 meets the threshold (“YES” at this operation), the method 270 goesto operation 284, in which the image process device 112 considers thatthe apparatus 102 is in a desired position (e.g., with a desired fieldof view) to capture a fundus image. As such, it is regarded that theimage includes the fundus and that the full field of view has beenachieved. Then, the image process device 112 can provide guideinformation to the user to inform that the apparatus 102 is in place(operation 286).

Referring to FIGS. 18-20, a method is described for providing guideinformation when the user moves the apparatus 102 relative to thesubject. In FIGS. 18-20, the images 330 illustrate example imagesdisplayed on the display screen such that the user can monitor theimages being captured by the apparatus 102 while moving the apparatus102 against the subject. The information displayed on the screen canassist the user to move the apparatus to a desired position andorientation.

In FIGS. 18 and 19, when the user locates the apparatus 102 completelyout of fundus (FIG. 18) or in fundus without a full field of view (FIG.19), a first mark 392 is activated, and a second mark 394 isdeactivated. The first mark 392 can be designed in various ways. In theillustrated example, the first mark 392 is configured as a circle in thecenter of the display screen. The first mark 392 can be activated ordeactivated in various ways. In some example, the first mark 392 changesits color when activated. For example, the first mark 392 remains ingray color or dimmed when deactivated, and changes to green whenactivated. Other color schemes or designs are also possible in otherexamples.

In FIG. 20, when the user locates the apparatus 102 into fundus with arelatively full field of view, the second mark 394 is activated, and thefirst mark 392 is deactivated. The second mark 394 can be designed invarious ways. In the illustrated example, the second mark 394 isconfigured as a diamond shape arranged at an upper side of the displayscreen. The second mark 394 can be activated or deactivated in variousways. In some examples, the second mark 394 changes its color whenactivated. For example, the second mark 394 remains in gray color ordimmed when deactivated, and changes to green when activated. Othercolor schemes or designs are also possible in other examples.

FIG. 21 illustrates an exemplary architecture of a computing device 400which can be used to implement aspects of the present disclosure,including the fundus image capture system 102 (or the image capturedevice 110, the image process device 112, the display device 114, theposition measure device 116, and/or the target guide device 118,separately), the server computing device 104, and the network 106, andwill be referred to herein as the computing device 400. The computingdevice 400 is used to execute the operating system, applicationprograms, and software modules (including the software engines)described herein.

The computing device 400 can be of various types. In some embodiments,the computing device 400 is a desktop computer, a laptop computer, orother devices configured to process digital instructions. In otherembodiments, the computing device 400 is a mobile computing device.Examples of the computing device 400 as a mobile computing deviceinclude a mobile device (e.g., a smart phone and a tablet computer), awearable computer (e.g., a smartwatch and a head-mounted display), apersonal digital assistant (PDA), a handheld game console, a portablemedia player, a ultra-mobile PC, a digital still camera, a digital videocamera, and other mobile devices.

The computing device 400 includes, in some embodiments, at least oneprocessing device 402, such as a central processing unit (CPU). Avariety of processing devices are available from a variety ofmanufacturers, for example, Intel or Advanced Micro Devices. In thisexample, the computing device 400 also includes a system memory 404, anda system bus 406 that couples various system components including thesystem memory 404 to the processing device 402. The system bus 406 isone of any number of types of bus structures including a memory bus, ormemory controller; a peripheral bus; and a local bus using any of avariety of bus architectures.

The system memory 404 includes read only memory 408 and random accessmemory 410. A basic input/output system 412 containing the basicroutines that act to transfer information within the computing device400, such as during start up, is typically stored in the read onlymemory 408.

The computing device 400 also includes a secondary storage device 414 insome embodiments, such as a hard disk drive, for storing digital data.The secondary storage device 414 is connected to the system bus 406 by asecondary storage interface 416. The secondary storage devices and theirassociated computer readable media provide nonvolatile storage ofcomputer readable instructions (including application programs andprogram modules), data structures, and other data for the computingdevice 400.

Although the exemplary environment described herein employs a hard diskdrive as a secondary storage device, other types of computer readablestorage media are used in other embodiments. Examples of these othertypes of computer readable storage media include magnetic cassettes,flash memory cards, digital video disks, Bernoulli cartridges, compactdisc read only memories, digital versatile disk read only memories,random access memories, or read only memories. Some embodiments includenon-transitory media.

A number of program modules can be stored in secondary storage device414 or memory 404, including an operating system 418, one or moreapplication programs 420, other program modules 422, and program data424.

In some embodiments, the computing device 400 includes input devices toenable a user to provide inputs to the computing device 400. Examples ofinput devices 426 include a keyboard 428, a pointer input device 430, amicrophone 432, and a touch sensitive display 440. Other embodimentsinclude other input devices. The input devices are often connected tothe processing device 402 through an input/output interface 438 that iscoupled to the system bus 406. These input devices 426 can be connectedby any number of input/output interfaces, such as a parallel port,serial port, game port, or a universal serial bus. Wirelesscommunication between input devices and interface 438 is possible aswell, and includes infrared, BLUETOOTH® wireless technology,802.11a/b/g/n, cellular, or other radio frequency communication systemsin some possible embodiments.

In this example embodiment, a touch sensitive display device 440 is alsoconnected to the system bus 406 via an interface, such as a videoadapter 442. The touch sensitive display device 440 includes touchsensors for receiving input from a user when the user touches thedisplay. Such sensors can be capacitive sensors, pressure sensors, orother touch sensors. The sensors not only detect contact with thedisplay, but also the location of the contact and movement of thecontact over time. For example, a user can move a finger or stylusacross the screen to provide written inputs. The written inputs areevaluated and, in some embodiments, converted into text inputs.

In addition to the display device 440, the computing device 400 caninclude various other peripheral devices (not shown), such as speakersor a printer.

The computing device 400 further includes a communication device 446configured to establish communication across the network. In someembodiments, when used in a local area networking environment or a widearea networking environment (such as the Internet), the computing device400 is typically connected to the network through a network interface,such as a wireless network interface 450. Other possible embodiments useother wired and/or wireless communication devices. For example, someembodiments of the computing device 400 include an Ethernet networkinterface, or a modem for communicating across the network. In yet otherembodiments, the communication device 446 is capable of short-rangewireless communication. Short-range wireless communication is one-way ortwo-way short-range to medium-range wireless communication. Short-rangewireless communication can be established according to varioustechnologies and protocols. Examples of short-range wirelesscommunication include a radio frequency identification (RFID), a nearfield communication (NFC), a Bluetooth technology, and a Wi-Fitechnology.

The computing device 400 typically includes at least some form ofcomputer-readable media. Computer readable media includes any availablemedia that can be accessed by the computing device 400. By way ofexample, computer-readable media include computer readable storage mediaand computer readable communication media.

Computer readable storage media includes volatile and nonvolatile,removable and non-removable media implemented in any device configuredto store information such as computer readable instructions, datastructures, program modules or other data. Computer readable storagemedia includes, but is not limited to, random access memory, read onlymemory, electrically erasable programmable read only memory, flashmemory or other memory technology, compact disc read only memory,digital versatile disks or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium that can be used to store the desired informationand that can be accessed by the computing device 400. Computer readablestorage media does not include computer readable communication media.

Computer readable communication media typically embodies computerreadable instructions, data structures, program modules or other data ina modulated data signal such as a carrier wave or other transportmechanism and includes any information delivery media. The term“modulated data signal” refers to a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, computer readable communication mediaincludes wired media such as a wired network or direct-wired connection,and wireless media such as acoustic, radio frequency, infrared, andother wireless media. Combinations of any of the above are also includedwithin the scope of computer readable media.

The computing device illustrated in FIG. 21 is also an example ofprogrammable electronics, which may include one or more such computingdevices, and when multiple computing devices are included, suchcomputing devices can be coupled together with a suitable datacommunication network so as to collectively perform the variousfunctions, methods, or operations disclosed herein.

Referring again to FIG. 21, the computing device 400 can include alocation identification device 448. The location identification device448 is configured to identify the location or geolocation of thecomputing device 400. The location identification device 448 can usevarious types of geolocating or positioning systems, such asnetwork-based systems, handset-based systems, SIM-based systems, Wi-Fipositioning systems, and hybrid positioning systems. Network-basedsystems utilize service provider's network infrastructure, such as celltower triangulation. Handset-based systems typically use the GlobalPositioning System (GPS). Wi-Fi positioning systems can be used when GPSis inadequate due to various causes including multipath and signalblockage indoors. Hybrid positioning systems use a combination ofnetwork-based and handset-based technologies for location determination,such as Assisted GPS.

As such, the fundus image capture system of the present disclosureprovides values that can be used to define if the user has found thereflection (e.g., IR reflection) from the pupil, the centroid (i.e.,target position) of the region of interest, and the current field ofview. This information can be used to guide the user through the funduscapturing process, thereby improving ease of use of the system. Themethods implemented in the system of the present disclosure are simpleand require less resource, thereby enabling fast image processing andevaluation.

The various examples and teachings described above are provided by wayof illustration only and should not be construed to limit the scope ofthe present disclosure. Those skilled in the art will readily recognizevarious modifications and changes that may be made without following theexamples and applications illustrated and described herein, and withoutdeparting from the true spirit and scope of the present disclosure.

What is claimed is:
 1. A method of capturing a fundus image, the methodcomprising: obtaining an image; applying a mask to the image to generatea masked image; identifying a fundus in the masked image; displaying themasked image on a display device; and provide guide information on themasked image to assist in capturing of the fundus image.
 2. The methodof claim 1, further comprising displaying a track point as the guideinformation.
 3. The method of claim 1, further comprising determiningthat a region of interest is identified in the masked image.
 4. Themethod of claim 1, further comprising eroding the masked image to removenoise.
 5. The method of claim 4, further comprising dilating the maskedimage.
 6. The method of claim 5, further comprising thresholding themasked image.
 7. The method of claim 6, further comprising applying abinary threshold to separate the masked image.
 8. The method of claim 1,further comprising dilating the masked image.
 9. The method of claim 1,further comprising thresholding the masked image.
 10. The method ofclaim 1, further comprising identifying a region of interest in themasked image.
 11. An apparatus for producing a fundus image, theapparatus comprising: a processor; an image capture device for capturingimages; and a non-transitory computer readable storage device storingsoftware instructions that, when executed by the processor, cause theapparatus to: obtaining an image from the image capture device; applyinga mask to the image to generate a masked image; identifying a fundus inthe masked image; displaying the masked image on a display device; andprovide guide information on the masked image to assist in capturing ofthe fundus image.
 12. The apparatus of claim 11, further comprisingsoftware instructions that, when executed by the processor, cause theapparatus to display a track point as the guide information.
 13. Theapparatus of claim 12, further comprising software instructions that,when executed by the processor, cause the apparatus to determine that aregion of interest is identified in the masked image.
 14. The method ofclaim 11, further comprising software instructions that, when executedby the processor, cause the apparatus to erode the masked image toremove noise.
 15. The method of claim 14, further comprising softwareinstructions that, when executed by the processor, cause the apparatusto dilate the masked image.
 16. The method of claim 15, furthercomprising software instructions that, when executed by the processor,cause the apparatus to threshold the masked image.
 17. The method ofclaim 16, further comprising software instructions that, when executedby the processor, cause the apparatus to apply a binary threshold toseparate the masked image.
 18. The method of claim 11, furthercomprising software instructions that, when executed by the processor,cause the apparatus to dilate the masked image.
 19. The method of claim11, further comprising software instructions that, when executed by theprocessor, cause the apparatus to threshold the masked image.
 20. Themethod of claim 11, further comprising software instructions that, whenexecuted by the processor, cause the apparatus to identify a region ofinterest in the masked image.